Systems which process image data for display come in many forms. These include video games which construct images from preexisting data, medical diagnostic systems which faithfully reproduce medical imaging data, and video decompression systems which decode encoded video information and process it to produce successive images for display. Each of these systems has a common component: a memory in which data representing image information is stored prior to display. Many image processing systems store more than one image, an input image and an output image. The input image may be processed, for example, to add visual features corresponding to a player of a video game, to electronically enhance components of a medical image, or for use in decoding a later occurring motion compensated encoded image.
Many image processing systems use block-oriented algorithms. These algorithms allow an image to be decomposed into smaller pieces (i.e. blocks) and the blocks to be processed individually. Significant time savings can be achieved by processing the pixels in an image block in parallel. Because they are processed in parallel, however, entire blocks of pixels may be available for storage and processing in a relatively short time interval.
Thus, memory systems used in block-oriented image processing apparatus should be able to fetch and store entire blocks of data quickly. This process may be complicated where block boundaries are not fixed. For example, when processing a motion compensated image the decoded data may represent differences between pixels in a block at one time and pixels in a block at a previous time. In this instance, pixels of the prior block are held in memory and fetched in order to complete the decoding process of the newly available pixels. Often, in order to obtain the best matching block of pixels, the prior block is taken from a different position in the image than that occupied by the newly received block. The boundaries of this reference block may be shifted with respect to the other blocks stored in the memory. Thus, the reference block may contain pixels from portions of several blocks in the stored image.
Other types of image processing such as interpolation may also use pixels from adjacent blocks to form a single block of the output image. One form of image interpolation which is commonly used for decoding a high definition video signal, is to define a reference block which is displaced by one-half pixel from a block that is stored in the image memory. A set of pixels which is larger than one block is accessed to regenerate the reference block having a resolution of one-half pixel position. Thus, the image memory should not be limited to accessing single blocks of pixels.
Another image processing problem which impacts on the structure and organization of an image memory is multi-component image data. For example, compressed video images may include separate luminance and chrominance components because chrominance information may be compressed to a greater degree than the luminance information without perceptively degrading the image. When an image is decoded and displayed, the luminance and chrominance components are processed together and displayed together. During image processing, however, it may be desirable to process the luminance components separately from the chrominance components. This puts added constraints on the image memory system, because luminance and chrominance components from separate blocks may need to be accessed together in some processing steps and separately in other processing steps.
An example of an image processing system which uses many of these techniques is an MPEG-2 decoder which decodes image data in 4:2:0 macroblock format. FIG. 1A is a graphical depiction of such an image which shows how the image is divided into slices. In FIG. 1A, each of the blocks labeled A through G is a separate slice of an image. Each slice is composed of many macroblocks. An exemplary macroblock is shown in FIG. 1B. This macroblock is in 4:2:0 format and so it has four eight-pixel by eight-pixel luminance blocks and two eight-pixel by eight-pixel chrominance blocks, one for the Cb color difference signal and the other for the Cr color difference signal.
When the image is received and decoded, the blocks occur in the sequence shown in FIG. 1C, that is to say, four luminance blocks, one Cb color difference block and one Cr color difference block. Image data is fetched from memory for display in the same order. In the display processor, each of the Cb and Cr blocks is expanded to four blocks and combined with respective ones of the four luminance blocks to regenerate a color image.
During image processing, however, the luminance and chrominance data may be accessed in blocks that do not conform to block boundaries. This is illustrated in FIGS. 1D and 1E. In FIG. 1D a reference block 110 is formed from parts of four other blocks 112, 114, 116 and 118. Thus, the boundaries of this reference block do not correspond to the boundaries of the image blocks that were stored in the memory when the image was decoded. Consequently, if the memory is arranged to access image blocks having fixed addresses, up to four image blocks may need to be accessed in order to regenerate the block 110.
FIG. 1E illustrates another image processing technique that is used in MPEG decoders. According to this technique a block 122 is regenerated, which not only does not align itself with image block boundaries, but does not align itself with pixel boundaries either. As shown in FIG. 1E, this block is displaced both horizontally and vertically by one half-pixel position from block 110, shown in FIG. 1D. To regenerate block 122, a number of pixels larger than is contained in a single block is accessed. This is illustrated by the nine pixel by nine pixel block 120 of FIG. 1E. As shown in FIG. 1E, it is desirable for an MPEG decoder to be able to access image data in blocks which are larger than the eight by eight pixel block size used by the MPEG-2 algorithm.